Multilingual Sentence Categorization according to Language

Issues in sentence categorization according to language is fundamental for NLP, especially in document processing. In fact, with the growing amount of multilingual text corpus data becoming available, sentence categorization, leading to multilingual text structure, opens a wide range of applications in multilingual text analysis such as information retrieval or preprocess-ing of multilingual syntactic parser. The major difficulties in sentence catego-rization are convergence and textual errors. Convergence since dealing with short entries involve discarding languages from few clues. Textual errors since documents coming from different electronic ways may contain spelling and grammatical errors as well as character recognition errors generated by OCR. We describe here an approach to sentence categorization which has the originality to be based on natural properties of languages with no training set dependency. The implementation is fast, small, robust and tex-tual errors tolerant. Tested for french, en-glish, spanish and german discrimination, the system gives very interesting results, achieving in one test 99.4% correct assignments on real sentences. The resolution power is based on grammatical words (not the most common words) and alphabet. Having the grammatical words and the alphabet of each language * This Paper is published in the Proceedings of the European Chapter of the Association for Computational Linguistics SIGDAT Workshop " From text to tags : Issues in Multilingual Language Analysis " held March 95 in Dublin. at its disposal, the system computes for each of them its likelihood to be selected. The name of the language having the optimum likelihood will tag the sentence — but non resolved ambiguities will be maintained. We will discuss the reasons which lead us to use these linguistic facts and present several directions to improve the system's classification performance. Categorization sentences with linguistic properties shows that difficult problems have sometimes simple solutions. Emergence of text categorization according to language came with the need of processing texts coming from all over the world. The goal of text categoriza-tion is to tag texts with the name of the language in which they are written. Information retrieval is the main application field. To do this job, the traditionnal way is to exploit the difference between letter combinations in different languages (Cavnar and Trenkle, 1994). For each language, the system computes from a training set a profile based on frequency (or probability) of letter sequences. Then, for a given text, it computes a profile and select the language which has the …